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PREFACE 


The  present  report  was  written  in  response  to  an  air  combat  training  Fleet 
Project  Team  (FPT)  requirement  that  a  performance  measurement  system  (PMS)  for 
the  Navy  Tactical  Aircrew  Training  System  (TACTS)  be  developed.  It  provides  a 


context  for  the  problem  and  value  of  performance  measurement  as  well  as  a  set  ) 
of  specific  functional  requirements  for  a  PMS  for  air-to-air  combat  training.  j 
In  addition,  the  report  provides  systematic  documentation  for  the  conclusion  r 


that,  with  the  development  of  the  extremely  impressive  TACTS  instrumentation 
technology,  there  was  no  parallel  development  to  provide  a  system  of  human 
performance  measurement  to  support  that  capability.  It  is  thought  that 
development  of  a  PMS,  which  meets  the  requirements  presented  here,  will 
greatly  assist  in  more  fully  accomplishing  the  original  operational  training 
requirements  which  led  to  development  of  the  TACTS  facility. 
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SECTION  I 
INTRODUCTION 


In  the  1950  Korean  conflict,  American  F-86  Sabre  jet  aircraft  destroyed 
equally  capable  Soviet  MIG-15  aircraft  at  a  rate  of  10  MiG's  for  every  Sabre 
lost  (Futrell,  1961).  In  view  of  these  results,  the  USAF  and  USN  were 
surprised  at  the  relatively  poor  exchange  ratios  of  approximately  2:1 
advantage  over  the  North  Vietnamese  Air  Force  during  the  first  half  of  the 
Vietnam  conflict  (DeLeon,  1977;  Editors,  AFJI ,  1974).  In  1969,  the  Navy 
released  the  results  of  a  study  directed  by  Captain  Frank  Ault  to  explain  the 
unexpected  poor  air  combat  maneuvering  (ACM)  performance  of  U.S.  pilots  in 
Southeast  Asia.  The  "Ault  Committee  Report"  (1969)  identified  deficiencies  in 
air  combat  training  as  a  primary  factor.  In  particular,  many  pilots  were 
reported  to  have  fired  their  weapons  (i.e.,  missiles)  outside  of  tactical 
launch  envelope  boundaries.  Failure  to  recognize  and  fire  within  an 
acceptable  geometric  cone  surrounding  a  target  aircraft  greatly  reduces  the 
probability  of  kill.  Because  the  Vietnam  Air  War  was  the  first  time  that  U.S. 
pilots  used  missiles  as  their  primary  air-launched  weapon  to  destroy  enemy 
aircraft  in  flight  (Craven,  1980),  inadequate  training  in  missile  envelope 
recognition  was  especially  apparent. 

In  response  to  the  Ault  Report,  the  Navy  developed  a  major  requirement 
that  a  designated  airspace  (range)  and  ground  support  be  designed  and 
implemented  to  provide  training  in  air  combat  skills.  The  specific  range 
requirements  were:  (1)  accurate  real  time  weapon  envelope  recognition 
training  during  ACM;  (2)  accurate  recording  of  training  events  for  debriefing, 
and  (3)  safety  and  economy  of  ACM  training  (Air  Test  and  Evaluation,  VX-4 , 
reported  in  Applied  Physics  Laboratory,  1975).  The  range,  originally  called 
the  Air  Combat  Maneuvering  Range  (ACMR),  is  today  known  as  the  Tactical 
Aircrew  Training  System  (TACTS).  The  general  system,  functional  description, 
and  operating  guidelines  for  TACTS  are  described  in  detail  elsewhere  (User's 
Guide,  1978). 
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SECTION  II 
PURPOSE 


Two  major  training  technologies  exist  to  provide  Navy  air-to-air  combat 
training.  The  first,  TACTS,  is  the  context  for  the  present  paper.  Following 
a  brief  description  of  the  TACTS  operational  and  training  capabilities,  the 
remaining  sections  of  the  paper  will  focus  on:  (1)  a  description  of  ACM 
performance;  (2)  an  analysis  of  the  human  performance  measurement  (PM) 
capabilities  and  limitations  of  TACTS;  (3)  an  assessment  of  the  value  of  TACTS 
performance  measures  for  a  variety  of  human  factors  applications;  (4)  an 
identification  of  different  approaches  used  to  conceptualize  and  measure  ACM 
performance;  and  (5)  development  of  a  set  of  specific  functional  requirements 
for  a  system  of  performance  measurements 

The  major  purpose  of  this  reporc  is  to  indicate  the  current  limitations  in 
Navy  TACTS  performance  measurement  relative  to  an  ideal  set  of  functional 
requirements  for  a  performance  measurement  system  (PMS).  The  major  thesis 
developed  herein  is  that  while  the  TACTS  original  design  included  performance 
measures,  there  was  no  specific  design  for  a  system  of  measurement.  It  is 
felt  that  development  of  a  PMS,  which  meets  the  requirements  presented  here, 
will  greatly  assist  in  more  fully  accomplishing  the  original  operational 
training  requirements  which  led  to  development  of  the  TACTS  facility. 

The  second  major  training  technology,  air  combat:  maneuvering  simulators 
(ACMSs),  are  somewhat  more  recent  innovations.  Two  classes  of  simulators  are 
used  for  Navy  ACM  training:  (1)  Weapon  System  Trainer,  Device  2F112,  located 
at  NAS  Miramar,  CA  and  NAS  Oceana,  VA;  and  (2)  Air  Combat  Maneuvering 
Simulator,  Device  2E6,  located  at  Oceana.^-  The  ACMSs  will  not  be  discussed 
specifically  below,  except  as  an  alternative  approach  to  TACTS  ACM  performance 
measurement . 


^■A  new  device,  Air  Combat  Tactics  Trainer,  2E7  ,  is  scheduled  to  be  ready 
for  training  at  NAS  Lemoore,  CA  in  late  1982,  and  a  second  unit  for  NAS  Cecil 
Field,  FL  in  mid-1983.  Youngling  et  al .  (1977)  provide  a  description  and 
evaluation  of  the  several  ACMSs  used  by  the  Air  Force  and  NASA. 
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SECTION  III 

DESCRIPTION  OF  TACTS  CAPABILITIES 


EXISTING  TACTS  OPERATIONAL  CAPABILITIES 

TACTS  is  an  advanced  air  combat  training  system  developed  to  improve 
aircrew  proficiency.  In  a  designated  airspace',  controlled  but  realistic  ACM 
missions  are  flown  against  U.S.  "play  to  kill”  aircraft  adversaries  that  mimic 
the  looks  and  tactics  of  Soviets  MIGs.  With  special  airborne  instrumentation 
pods,  all  significant  flight  parameters,  weapon  events,  and  mission  data  of 
multiple  aircraft  are  transmitted  to  ground  stations  where  they  are  displayed 
in  real  time  to  the  range  training  officer  and  recorded  for  replay  during  post 
mission  debriefing.  Tracks  of  the  symbolized  aircrafc  are  presented  on  a 
three-dimensional  (3-D)  Cathode  Ray  Tube  (CRT)  display.  The  system  also 
.  a-timizes  safety  of  training.  If  an  aircraft  violates  any  preprogrammed 
safety  parameter,  the  pre-designated  symbol  for  that  aircraft  begins  flashing 
on  the  display  screen.  The  ground  instructor  can  then  warn  the  pilot.  Figure 
1  portrays  the  four  subsystems  comprising  TACTS. 


Figure  1.  Description  of  the  Four  Subsystems  of  the  Tactical  Air  Combat 
Training  System  (TACTS).  Description  is  from  the  East  Coast 
TACTS  User's  Guide  (May  1978) 


^The  Air  Force  participated  in  the  evaluation  of  the  first  Navy  range 
and  established  a  subsequent  requirement  for  an  Air  Force  range  called  the  Air 
Combat  Maneuvering  Instrumentation  (ACMI)-  The  first  ACMI  became  operational 
in  July  1976  at  Nellis  AFB,  NV,  as  a  technologically  updated  version  of  the 
TACTS . 
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The  first  TACTS  became  operational  m  December  1973  at  MCAS  Yuma,  A2^  . 
Ground  stations  for  monitoring  and  debriefing  ACM  in  the  Yuma  range  are 
available  at  both  Yuma  and  NAS  Miramar,  On  the  east  coast,  an  over-water 
>*ange  became  operational  in  1976  off  Jape  Hatteras,  NC •  Ground  training 
facilities  are  available  at  both  NAS  Oceana,  and  MCAS,  Cnerry  Point,  NC. 

EXISTING  TACTS  TRAINING  CAPABILITIES 

The  TACTS  training  capabilities  provide:  (1)  highly  realistic  conditions 
approximating  actual  combat;  (2)  management,  and  dirc-t  contro]  of  training  by 
ground-based  instructors  who  can  quickly  vector  participating  aircraft  into 
positions  to  set  up  for  each  engagement;  (3)  several  missile  envelope 
recognition  training  modes,  varying  progressively  in  difficulty;  (4)  immediate 
knowledge  i  f  results  of  simulated  weapon  release  outcomes;  (5)  "nondebatable" 
(i.c.,  obj  ctive)  magnetic  tape  record  of  certain  actions  in  the  aircraft;  and 
(6)  recort  d  flight  and  weapon  action  data  for  performance  debriefing  and 
hardcopy  jedback. 

EMERGING  TACTS  TRAINING  CAPABILITIES 

The  n.'.me  change  from  ACMR  to  TACTS  in  1980  reflects  the  broadening  of 
anticipat id  training  system  capabilities  beyond  air  combat  maneuvering  to  a 
variety  aircrew  combat  tasks.  Emerging  caDabilities  include  simulated 
air-tr -ground  weapons  delivery  (e.g.,  no-drnp  bomb  scoring),  electronic 
warfare  (EW)  training,  and  surface-to-air  missile  (SAM)  avoidance  training. 
Initial  studies  have  been  completed  to  investigate  the  feasibility  of  adding 
by  FY36  ;>t-sea  monitoring  and  debriefing  capabilities  aboard  aircraft  carrier 
battle  groups  to  the  currently  available  land-based  facilities  (Crangle, 

1980) , 3 


^Current  ranges  will  also  be  modiried  as  needed  lor  operational  test  and 
evaluation  (OT&E)  research  studies  such  as  the  Air  Combat  Evaluation  -  Air 
Intercept  Missile  Evaluation  ( ACEVAL-AIKVAL)  program  -  a  joint  Air  Force/Navy 
evaluation  of  new  air  intercept  missiles  ,nd  t3ctics. 
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SECTION  IV 

DESCRIPTION  OF  ACM  PERFORMANCE 


The  basic  ACM  task  appears  simple.  The  pilot  attempts  to:  (1)  destroy 
the  enemy;  and  (2)  avoid  being  destroyed  by  the  enemy.  According  to  John 
Johnson  (1956),  Britain's  top  fighter  ace  in  WWII,  pilots  just  before  an  ACM 
mission.  .  .  "fall  into  two  broad  categories;  those  who  are  going  to  shoot  and 
those  who  secretly  and  desperately  know  that  they  will  be  shot  at  .  .  .”  He 
called  the  first  group  "hunters"  and  the  second  group  "hunted."^  Tradi¬ 
tionally,  however,  the  fighter  pilot  has  been  identified  with  the  first, 
aggressive  role  and  to  a  much  lesser  extent  with  the  second,  survival  role. 

The  cultural  mystique  surrounding  the  aggressive  fighter  pilot  and  the  nature 
of  his  combat  task  are  described  below. 

TASK  COMPLEXITY 

During  ACM  the  modern  pilot  must  visually  detect,  identify,  and  track  an 
adversary  which  can  change  position  in  3-D  geometric  space  at  supersonic 
speeds.  Meanwhile,  the  fighter  must  also  rapidly  manipulate  his  aircraft  and 
weapon  controls,  monitor  instruments  and  displays,  and  coordinate  his  actions 
over  communication  channels  with  other  aircrew  and/or  friendly  fighter 
aircraft.  All  these  dynamic  responses  must  take  place  within  the  typically 
short,  2-3  minutes,  of  an  ACM  engagement.  In  addition  to  the  stringent 
perceptual-motor  shill  requirements,  decision-making  demands  are  continuous. 
For  example,  missile  envelope  recognition  is  a  critical  decision  based  on  a 
number  of  factors,  including  the  of  aircraft  adversary,  fighter  missile 

typa  selected,  various  rules  of  thumb  for  envelope  determination,  etc. 

These  perceptual-motor  and  decision-making  skills  are  "enabled”  only 
through  the  mastery  of  a  third  skill-control  over  emotional  responses. 5  The 
pilot  must  keep  his  "cool"  under  conditions  of  high  gravitational  forces,  the 
possibility  of  a  ground  or  midair  crash,  and  the  extremely  rapid  loss  of  fuel 
during  intense  ACM  on  the  TACTS.  Du' Ing  actual  combat,  the  pilot  contends 
with  additional  stressors,  including  the  likelihood  of  multiple  adversaries 
with  real  weapons,  communicaton  jamming,  SAM  missiles  and  enemy  "flak"  of 
various  kinds.  Although  98  percent  of  pilots  surveyed  have  highly  favorable 
opinions  oi  the  instrumented  ACM  range  as  a  training  facility  (Youngling 


^Weiss  (1966)  dichotomized  combat  pilots  into  "Hawks"  (those  that  downed 
enemy  aircraft)  and  "doves"  (those  that  were  shot  down  by  enemy  aircraft). 

^A  collaborative  effort  between  the  Naval  Health  Research  Center/Naval 
Training  Equipment  Center  (NAVTRAEQUIPCEN)  is  planned  for  FY82  to  identify 
training  "stress  profiles"  of  pilots  during  the  experience  of  ACM  and  to 
relate  these  profiles  to  objective  measures  of  ACM  performance. 
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et  al.,  1977),  there  was  also  consensus  that  the  relative  safety  of  the 
training  environment  limited  to  some  degree  transfer  of  training  to  the 
life-threatening  episodes  of  combat  ACM. 

Air-to-air  combat  between  aircraft  is  especially  complex  because  in-close 
aircraft  maneuvering  is  required.  The  classical  and  modern  "dogfight"  or 
"hassle"  between  a  fighter  and  an  adversary  target  (called  a  "bogey")  takes 
place  because  of  a  requirement  for  visual  identification  (VID)  of  an  adversary 
prior  to  weapon  release.  The  operational  requirement  for  VID  lessens  the 
likelihood  of  destroying  friendly  aircraft.  However,  VID  is  not  necessary’  for 
existing  missile  effectiveness  which  can  extend  well  beyond  visual  acquisition 
range,  and  beyond  the  airspace  in  which  visually  controlled  dogfighting  would 
otherwise  occur. ^ 

The  abbreviation  "ACM"  is  often  used  as  a  catch-all  expression  for  the 
entire  air-to-air  combat  mission  which  is  highlighted  by  the  aircraft 
maneuvering  engagement,  (dogfight).  The  entire  mission  includes  several  kinds 
of  performance  skills,  including  radar  procedures,  VID  intercept  procedures, 
tactics,  ACM,  weapon  system  and  missile/cannon  envelope  recognition,  and 
"bugout"  (return  to  base)  procedures. 

THE  "FIGHTER  MYSTIQUE" 

Until  recently,  attempts  to  quantify  the  highly  complex  ACM  task  and  to 
reduce  it  to  elemental  sequences  of  subskills  were  scarce.  The  ACM 
performance  measurement  (ACKPM)  methods  described  below  are  all  recent 
products  of  the  last  decade.  Part  of  the  historical  reluctance  to  develop 
ACMPM  systems  can  be  attributed  to  what  might  be  called  the  "fighter  mystique" 
(Sddowes,  1981).  There  appears  to  be  a  general  sentiment  in  the  populace  that 
fighter  pilots  have  the  "right  stuff"  (Wolfe,  1980).  It  is  also  generally 
acknowledged  that  the  ingredients  for  the  right  stuff  are  complex  and 
unknown.  Even  if  the  ingredients  were  known,  they  could  not  be  quantified. 
Finally,  the  argument  continues,  even  if  they  could  be  quantified,  so  few 
people  would  have  the  right  stuff  that  measurement  would  be  impractical. 

Studies  of  combat  reviewed  by  Youngling  et  al.  (1977)  support  the  general 
view  that  only  a  few  fighter  pilots  have  the  right  ingredients  for  combat 
success.  For  example,  only  five  percent  of  the  5,000  fighter  pilots  who  flew 
against  the  Germans  in  WWII  during  1943-1945  accounted  for  40  percent  of  the 
kills  recorded  by  t.he  Eighth  Air  Force  during  that  period.  Even  more 
impressive  were  the  German  fighters  whose  ten  best  chalked  up  2,568  kills 
among  them  (Weiss,  1966). 


^Certain  navy  combat  experienced  ACM  pilots  (e.g.,  Flym,  1975)  feel 
that  the  advent  of  new  missile  technology  "will  not  eliminate  the  need  for 
visual  identification  of  enemy  aircraft  and  hence  the  inevitable  dogfight"  (p. 
4).  An  identical  view  has  jeen  expressed  by  Air  Force  pilots  (Efhell,  1980). 
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The  unfortunate  consequence  of  the  fighter  mystique  attitude  present  in 
this  culture  is  that  the  skill  components  of  the  ACM  task  have  not  been 
translated  into  performance  measures  which  could  be  used  to  provide  training 
feedback  for  the  majority  of  fighter  pilots  who  could  profit  from  such 
feedback.  Data  from  WWI,  WWII  and  the  Korean  conflict  indicate  that  less  than 
15  percent  of  all  fighter  pilots  had  a  better  than  even  chance  of  surviving 
their  first  combat  experience  (Weiss,  1966).  Based  on  data  such  as  these,  it 
has  been  estimated  that,  without  specialized  air-to-air  combat  training,  high 
attrition  rates  can  be  expected  in  future  conflicts  (Youngling  et  al.,  1977). 
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SECTION  V 

EXISTING  TACTS  ACM  PERFORMANCE  MEASUREMENT  (ACMPM) 
CAPABILITIES  AND  LIMITATIONS 


As  described  above,  TACTS  contributes  directly  to  training  by  providing  an 
ACM  performance  measurement  capability.  TACTS  records  and  plays  back 
objective  performance  data  in  a  variety  of  graphic  and  numeric  formats. 
However,  the  existing  TACTS  ACMPM  has  a  number  of  specific  limitations  for 
training  which  will  be  described  below.  The  full  potential  of  ACMPM  has  yet 
to  be  realized.  As  evidence  of  the  need  for  a  system  of  performance 
measurement  on  the  TACTS,  the  Navy  Fleet  Project  Team  (FPT)  for  air  combat 
training,  recently  stated: 

"A  performance  measurement  system  (PMS)  is  required  to  determine  training 
effectiveness,  act  as  a  diagnostic  training  aid,  and  evaluate  present/ 
future  training  device  capabilities  relative  to  the  syllabus.  As  a  basis 
for  PMS,  indices  must  be  identified.  A  standard  PMS  should  be 
established."  (COMTACWINGSLANT  naval  message,  March,  1981.) 

LIMITATIONS  IN  EXISTING  ACMPM 

The  limitations  indicated  below  confirm  the  conclusions  of  the  FPT  and 
provide  a  foundation  for  the  development  of  a  set  of  requirements  for  an 
ACMPMS  in  support  of  the  fleet. 

DATA  FLOODING.  The  performance  measures  available  during  ACM  and  for  the  ACM 
debrief  are  so  "rich"  that  some  users  have  described  the  scheme  as  "data 
flooding".  The  instructor  is  genuinely  overloaded  with  such  an  array  of 
information. 

LACK  OF  SPECIFIC  TRAINING  OBJECTIVES.  Data  flooding  is  a  natural  consequence 
of  inadequate  specification  of  behavioral  objectives.  When  it  is  uncertain  as 
to  what  is  being  trained,  the  simple  (and  often  more  costly)  solution  is  to 
measure  everything. 

LACK  OF  TREND  DATA.  A  major  limitation  of  the  TACTS  is  that  performance  data 
are  not  accumulated  over  engagements  to  identify  aircrew  trends  (Ciavareili, 
Williams,  &  Stoffer,  1981).  Trend  data  are  necessary  to  identify  and  to 
diagnose  consistent  patterns  of  performance  as  a  baseline  for  corrective 
action. 

CERTAIN  IMPORTANT  VARIABLES  UNMEASURED.  A  second  consequence  of  inadequate 
specification  of  training  objectives  is  that  an  important  performance  variable 
may  go  unmeasured.  One  such  variable,  time  in  envelope,  is  not  available 
under  the  present  TACTS  PM.  This  omission  is  interesting  since  envelope 
recognition  skill  training  was  one  of  the  three  original  requirements  leading 
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to  development  of  TACTS. ^  A  second  variable,  energy  .-anagemenf ,  nas  only 
recently  been  available  as  a  debriefing  cid  at  NAS  Miramar.  An  energy 
management  display  (EMD)  measures  the  maneuvering  capability  of  the  aircraft 
by  integrating  energy  maneuverability  di.ta  available  from  the  TACTS.  The 
ability  to  maximize  maneuvering  capability  is  a  major  determinant  of  the 
outcome  of  an  owbat  (Deberg,  1977;  Pruitt,  Moroney,  &  Lau,  1980). 

INADEQUATE  DEBRIEF  DATA  FORMATTING.  Much  of  the  performance  data  available 
during  the  debrief  is  in  numeric  form  which  is  difficult  to  visually  process 
and  retain.  The  operational  acceptance  of  the  EMD  is  based  in  large  part  on 
the  fact  that  it  is  a  pictorial  display  of  the  energy  maneuvering  envelope  of 
an  aircraft  and  of  its  opponent  during  an  ACM  engagement.  The  EMD  was 
developed  as  a  debriefing  tool  with  considerable  input  from  TACTS  pilots. 
Similarly,  the  complexity  of  missile  envelopes  would  be  more  easily  understood 
if  represented  by  graphic  than  numeric  formats.  An  ongoing  project,  directed 
by  the  Human  Factors  Laboratory  (HFL)  at  the  Naval  Training  Equipment  Center, 
has  focused  on  the  development  of  methods  to  standardize  the  entire  debrief 
feedback  in  terms  of  display  tables  and  graphs  that  are  recommended  by  TACTS 
users  (Ciavarelli,  Pettigrew,  &  Brictson,  1981).  Methods  for  missile  envelope 
representation,  and  simultaneous  representation  of  associated  missile  shots, 
are  now  available. 

LACK  OF  QUALITY  CONTROL  OVER  RAW  PERFORMANCE  DATA.  Two  major  emerging  systems 
to  provide  performance  measurement  for  TACTS  will  be  described  in  a  subsequent 
section.  Both  of  these  efforts,  as  well  at  a  more  recent  effort  with  a 
similar  goal  (McGuinness  et  al.,  1980),  have  had  to  implement  considerable 
quality  control  over  the  performance  data  prior  to  data  analysis  and 
interpretation.  McGuinness  et  al.  have  described  in  some  detail  a  number  of 
limitations  in  TACTS  hardware  and  operational  procedures  which  consequently 
require  "filtering”  of  TACTS  output  data. 


^Even  if  a  time  in  envelope  indication  were  immediately  available,  it 
would  probably  be  necessary  to  reconstruct  or  qualify  the  variable  (i.e.,  on 
the  basis  of  a  rate  measure). 
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SECTION  VI 

VALUE  OF  TACTS  AS  A  SOURCE  OF  PERFORMANCE  CRITERIA 


The  objective  performance  data  available  from  TACTS  are  sources  of 
valuable  criteria  for  several  purposes,  including  tr  lining. 

SELECTION 

One  of  the  original  (1975)  research  interests  in  the  TACTS  ACMPM  was  to 
identify,  define  and  validate  behavioral  criterion  variables  for  use  by  the 
Naval  Aerospace  Medical  Research  Laboratory  (Brictson,  Ciavarelli  &  Jones, 

1977;  Brictson  et  al . ,  1978).  The  Vision  Research  Laboratory,  in  particular, 
was  interested  in  developing  realistic  criteria  for  validation  of  certain 
visual  variables  used  to  select  naval  aviators.  There  appeared  to  be  an 
obvious  mismatch  between  current  selection  variables  which  emphasize  static, 
high  contrast,  central  vision  acuity  and  the  dynamic,  low  contrast,  peripheral 
vision  characteristic  of  ACM  -  a  major  mission  of  the  navy  fighter  pilot 
community.  In  a  study  using  TACTS  data,  e.g.,  it  was  found  that  the  average 
visual  target  acquisition  ("tally-ho")  range  was  considerably  shorter  than 
expectation  based  on  laboratory  data  on  human  visual  capability  (Hutchins, 
1978).  These  laboratory  data  are  the  basis  for  the  visual  selection  tests 
currently  in  use  by  the  Navy.  Youngling  et  al.  (1977)  have  systematically 
reviewed  and  recommended  ACM  ranges  as  intermediate  test  beds  for  combat 
effectiveness  predicted  by  various  selection  factors. 

TRAINING 

Performance  data  are  essential  to  the  training  process.  Measures  of  ACM 
subtask  skills  allow  the  instructor  to  monitor  the  progress  of  training,  as 
well  as  to  provide  diagnostic  feedback  regarding  problem  areas.  Both 
norm-referenced  and  criterion-referenced  standards  of  aircrew  proficiency  can 
be  established.  These  behavioral  criteria  and  associated  standards  inform  the 
instructor  and  student  as  to  what  is  to  be  trained  and  to  what  level. 

Training  effectiveness  evaluations  (TEEs)  are  then  possible. 

ASSESSMENT 

Assessment  of  operational  ACM  readiness,  tactics,  equipment,  safety,  and 
cost-effectiveness  of  the  range,  is  possible  only  with  the  availability  of 
TACTS  performance  data. 

READINESS  ASSESSMENT.  Once  combat  becomes  necessary,  the  outcome  rests  on  the 
quality  of  the  previous  aviator  selection  process  and  training  in  ACM.  A 
third  factor  is  placement  or  deployment  of  aircrew  and  squadrons  on  missions 
that  are  suited  to  their  level  of  operational  readiness.  Readiness  assessment 
is  not  concerned  with  changing  the  level  of  ACM  proficiency  through  training; 
it  simply  attempts  to  describe  existing  squadron  and  individual  differences 
among  fleet  fighters. 
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TACTICS  ASSESSMENT.  Performance  data  taken  during  TACTS  engagements  can 
update  ACM  tactics.  The  Naval  Fighter  Weapons  School  (NAVFITWEP3C0L) .  called 
"Topgun,"  is  responsible  for  teaching  advanced  ACM  tactics.  Lessons  learned 
from  tactics  assessment  on  the  TACTS  have  already  begun  to  update  Topgun 
tactical  guidance,  including  those  that  concern  throttle  control  during  ACM 
and  various  "rules  of  thumb"  for  recognizing  missile  launch  envelopes. 

EQUIPMENT  ASSESSMENT.  The  technology  of  TACTS  is  changing  rapidly  with  the 
development  and  introduction  of  new  aircraft  (i.e.,  the  F/A-18),  new  air 
intercept  missiles  (i.e.,  all  aspects  capable),  new  performance  aids 
incorporated,  in  Heads-Up-Display  (HUD)  to  improve  missiJe  envelope  recognition 
(Lucter,  1979),  new  EMD  performance  aids  developed  to  maximize  the  maneuvering 
capability  of  the  fighter  aircraft  (Pruitt  et  al.,  1980),  etc.  Performance 
criteria  will  serve  to  evaluate  whether  the  use  of  modified  capabilities  and 
performance  aids  translates  into  improved  performance  on  TACTS. 

SAFETY  ASSESSMENT.  Similar  to  commercial  airline  recording  of  flight  data  and 
human  performance  for  subsequent  analysis  for  safety  factors,  TACTS  allows 
both  on-line  and  off-line  capability  for  safety  assessments  leading  to 
corrective  and/or  preventive  feedback. 

COST-EFFECTIVENESS  ASSESSMENT.  The  use  of  simulated  weapons  and  targets  has 
greatly  decreased  the  cost  (and  increased  the  safety)  of  ACM  practice.  The 
producer  of  the  instrumented  air  combat  ranges  for  both  the  Navy  and  the  Air 
Force  estimates  that  the  TACTS/ACMI  reduces  air  combat  training  costs  by  more 
than  $100  million  annually  (Cubic  Corporation,  1978).  According  to  Bill 
Dollard  (1980),  the  TACTS  manager  at  Miramar,  the  fir.it  system  developed  for 
Miramar  cost  approximately  $25  million  in  R&D  and  installation  of  the  cystem 
in  an  operational  mode.  The  system  costs  less  than  $1,000  an  hour  to 
operate.  However,  these  savings  must  be  assessed  in  terms  of  performance 
effectiveness  measures  on  the  TACTS. 

SIMULATION  DESIGN 

One  of  the  most  important  values  of  PM  is  its  role  in  training  device 
design.  Future  Navy  air  combat  training  simulators  will  be  designed  with  the 
benefit  of  TACTS  performance  data  that  has  validated  certain  ACM  simulator 
training  capabilities  and  not  others.  In  particular,  the  simulator's  PMS  can 
be  designed  to  overlap  that  available  on  the  TACTS.  Common  performance 
measures  between  ACM  trainers  and  TACTS  would  provide  an  ideal  foundation  for 
effectiveness  evaluations  of  simulators  (see  McGuinness  et  al.,  1980). 
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SECTION  VII 

MAJOR  APPROACHES  TO  ACM  PERFORMANCE  MEASUREMENT'  (ACMPM) 


The  literature  on  ACMPM  can  be  organized  on  the  basis  of  common  approaches 
to  the  problem  of  measurement. 

SYSTEMS  VIEW  VS.  COMPONENT  VIEW 

Most  researchers  in  this  field  have  taken  a  systems  view  of  ACMPM.  Thus, 
human  performance  and  its  measurement  is  viewed  as  only  one  component 
dependent  on  other  elements  in  an  ACM  training  system.  The  system  includes 
such  elements  as  aircrew,  instructors,  aircraft,  weapons,  and  operating 
conditions,  all  of  which  impact  performance  output. 

MULTIVARIATE  VS.  UNIVARIATE  ANALYSES 

Most  researchers  have  recognized  the  complexity  of  ACM  success  and  have 
adopted  various  statistical  techniques  to  deal  with  this  complexity.  Not 
surprisingly,  multivariate  methods  such  as  multiple  regression  (Coward  et  al., 
1979),  factor  analysis  (Deberg,  1377),  and  discriminant  function  analysis 
(Kelly  et  al.,  1979),  have  been  applied  primarily  with  ACM  simulators  rather 
than  with  the  TACTS.  In  contrast  to  TACTS,  these  simulators  allow  sufficient 
pilot  sample  size  and  repeated  measures  on  selected  variables  to  allow  the  use 
of  multivariate  statistical  approaches  to  the  development  of  ACMPM. 

The  Readiness  Estimation  System  (RES),  described  below,  provides  a 
performance  index  of  ACM  based  on  complex  mathematical  models  that  incorporate 
a  number  of  airplane  and  inter-airplane  parameters  typically  measured  by  the 
TACTS. 

It  should  be  noted  that  operational  users  (Seminar,  1980)  of  TACTS  are 
generally  opposed  to  a  simple,  univariate  characterization  of  performance. 
Their  legitimate  fear  is  that  ACM  training  would  be  geared  to  achievement  of 
that  particular  performance  metric  to  the  exclusion  of  others  critical  for  ACM 
proficiency.  In  addition,  there  appears  to  be  a  desire  to  avoid  labeling  the 
performance  of  any  particular  pilot  based  on  one  measure  that  might  have 
adverse  effects  on  his  competitive  spirit.  Resolution  of  the  apparent 
conflict  between  research  development  of  single  metrics  and  the  concerns 
identified  above  will  be  found,  in  part,  by  development  of  a  composite  measure 
of  ACM  success,  dependent  on  multiple  tasks,  with  each  task  weighted  according 
to  its  relative  contribution  to  overall  ACM  success.  This  composite  measure, 
under  development  by  the  NAVTRAEQUIPCEN  Human  Factors  Laboratory,  will  allow 
both  subtask  and  composite  ACM  task  performance  feedback,  to  supplement  the 
traditional  measure  of  ACM  success  in  terms  of  fighter/adversary  kill  exchange 
ratios . 

"BUILDING  BLOCKS"  VS.  "TOP-DOWN"  VIEW  OF  ACM  SUCCESS 

Some  (e.g.,  Ciavarelli,  Williams,  &  Krasovec,  1980)  have  chosen  a 
hierarchical  or  "building  blocks"  system  approach  to  development  of  a  FMS  by 
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first  focusing  on  specific  ACM  tasks  that  are  evaluated  at  various  layers  in 
the  performance  hierarchy  for  criticality  to  ACM  success.  Others  use  a 
"top-down"  approach  by  focusing  on  ultimate  or  penultimate  performance 
measures  of  ACM  success  and  then  proceeding  to  subtask  analysis.  For  example, 
McGuinneso  et  al.  (1980)  have  sharply  criticized  approaches  based  strictly  on 
subtask  analysis.  At  some  level,  however,  agreement  between  these  two 
approaches  will  eventually  be  reached. 

Performance  variables  at  different  levels  in  the  measurement  hierarchy  are 
useful  for  different  purposes.  For  example,  task  and  subtask  approaches 
provide  sufficient  detail  for  individual  pilot  diagnostic  feedback.  At  this 
level  it  is  possible  to  find  out  what  specific  skills  superior  fighter  pilots 
possess  so  that  these  skills  can  be  trained  in  new  pilots.  Approaches  that 
focus  on  system  output  level  such  as  exchange  ratios  provide  valuable 
information  for  overall  squadron  readiness  assessment.  The  RES,  e.g., 
provides  overall  ranking  of  aircrew  and  squadrons  on  the  basis  of  overall 
maneuvering  scores,  but  does  not  allow  for  individual  diagnostic  information 
to  explain  differences  in  maneuvering  proficiency.  At  the  system  level  are 
output  variables  that  are  used  to  assess  the  relative  contribution  of  various 
subtasks  and  to  construct  a  composite  measure  of  ACM  success  referred  to 
above.  Thus,  not  only  do  performance  measures  at  various  levels  in  the 
hierarchy  serve  useful  purposes  by  themselves  but  they  also  complement  each 
other  in  a  complete  system  of  performance  measurement. 

DYNAMIC  VS.  STATIC  REPRESENTATIONS  OF  PERFORMANCE 

The  original  TACTS  provides  qualitative  feedback  through  dynamic  replay  of 
the  entire  engagement.  Some  more  recent  PM  schemes  (i.e.,  the  RES)  provide 
performance  feedback  regarding  the  dynamic  interplay  between  opposing 
aircraft.  The  RES  provides  a  time-history,  quantitative  index  score  of 
maneuvering  performance  based  on  continuous  measurement  throughout  the 
engagement . 

Other  approaches  conceptualize  maneuvering  as  one  rather  discrete  step  in 
a  total  ACM  mission  sequence  preceded  by  radar  procedures  and  lookout 
procedures,  and  followed  by  envelope  recognition  procedures  (Ciavarelli  et 
al.,  1981).  Obviously,  the  dynamic  properties  within  each  of  these,  discrete 
"steps"  could  be  elaborated  into  a  PM  metric.  For  training  diagnostic 
purposes,  however,  there  is  considerable  value  in  providing  initial 
performance  feedback  in  the  form  of  discrete  and  comprehensible  steps  Involved 
in  the  task  to  be  trained.  The  more  complex  and  less  comprehensible 
performance  dynamics  associated  with  each  step  should  be  reserved  until  the 
pilot  has  achieved  minimal  proficiency  at  those  particular  steps.  In  reality, 
the  distinction  between  dynamic  and  static  representations  of  performance 
becomes  blurred  to  the  extent  that  sequential  dependencies  between  discrete 
events  in  the  ACM  mission  can  he  established,  as  has  been  demonstrated  by 
Ciavarelli  and  his  associates. 
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PRESELECTED  PILOTS  VS.  PRESELECTED  VARIABLES 

Most  research  in  ACMPM  falls  into  one  of  two  approaches.  One  approach 
(e.g.f  Kelly  et  al . ,  1979)  first  preselects  pilots  high  or  low  in  overall  ACM 
proficiency,  on  the  basis  of  instructor  ratings,  simulator  performance,  or 
past  TACTS  records.  These  two  groups  are  then  contrasted  on  the  basis  of  a 
large  number  of  specific  performance  measures.  Measures  that  differentiate 
the  two  groups  are  prime  candidates  for  the  PM  scheme.  The  second,  and  more 
common  approach,  preselects  a  limited  number  of  specific  variables  for 
possible  inclusion  into  the  PM  scheme  and  then  tests  them  for  relative 
reliability  and  validity  across  a  wide  range  of  pilot  proficiency  levels. 

AIR  COMBAT  MANEUVERING  SIMULATOR  (ACMS)  VS.  TACTS 

Ideally,  the  design  of  ACM  simulators  and  their  associated  PM  system  is 
preceded  by  development  of  a  PM  system  validated  on  the  TACTS  range  itself. 
Unfortunately,  none  of  the  existing  Navy  ACMSs  has  benefitted  from  a  TACTS 
PMS.  Thus,  ACM  simulators  have  been  recently  used  not  only  to  develop  ACM 
FMSs  for  use  with  the  simulators  themselves,  but  also  as  a  major  approach  for 
developing  a  possible  PMS  for  TACTS.  Through  an  iterative  process,  there  will 
eventually  be  developed  a  set  of  measures  common  lo  the  ACMS  and  TACTS. 

In  addition  to  the  obvious  benefits  of  increased  safety  and  economy  in  an 
opertaional  training  area  traditionally  characterized  by  high  risk  and  high 
cost,  ACMSs  allow  pilots  to  experience  a  wide  variety  of  tactical  situations 
and  gain  more  practice  with  these  situations  than  is  currently  possible  on  the 
TACTS.  Thus,  with  the  ACMS,  it  is  feasible  to  actively  control  training 
experience  and  to  achieve  repeated  measures  of  performance  data  -  both  of 
which  are  essential  for  PMS  development  and  extremely  difficult  with  the 
TACTS.  The  only  systematic  attempt  to  develop  an  ACMS  PMS  with  the  Navy  is 
just  beginning,  with  the  2E6  at  Oceana,  under  the  direction  of  the 
NAVTRAEQUIPCEN  Human  Factors  Laboratory.® 


®There  are  two  other  emerging  and  continuing  Air  Force  ACMS  PMSs  -  one 
called  Tactical  Space  (TACSPACE)  under  joint  AFHRL/NAVTRAEQUIPCEN  sponsorship 
(Kelly  et  al.,  1979),  and  one  called  the  Good  Stick  Index  (GSI)  developed  by 
Coward  et  al.  (1979). 
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SECTION  VIII 

FUNCTIONAL  REQUIREMENTS  FOR  AN  ACMPM  SYSTEM 


In  view  of  Che  above  described  limitations  of  original  TACTS  PMS,  the 
recenc  FPT  message  stating  the  need  to  develop  a  PM  system  for  use  with  the 
TACTS,  and  the  tremendous  potential  value  of  TACTS  performance  criteria,  it  is 
clear  that  functional  requirements  for  a  PM  system  should  be  identified.  As 
indicated  previously,  a  system  of  ACM  measurement  is  yet  to  be  developed. 

The  purpose  of  this  section  is  to  identify  several  psychometric,  training, 
and  user  feasibility  requirements  that  must  be  met  for  a  PMS  to  be  developed. 
These  functional  requirements  will  be  briefly  described  using  two  emerging 
TACTS  ACMPMSs.  Both  systems  use  the  original  TACTS  ACMPM  as  a  basis  for 
elaboration.  Both  emphasize  highly  automated  methods  to  manage  the  TACTS 
outputs.  It  is  not  intended  to  comprehensively  compare  the  two  or  to 
recommend  either  one  or  the  other  PMS.  If  anything,  both  systems  are 
recommended  for  further  development.  They  were  chosen  to  illustrate  PMS 
requirements  because  they  are  the  only  systems  that:  (1)  have  been  under 
development  for  several  years;  (2)  meet  many  of  the  requirements  identified 
below;  (3)  have  a  considerable  body  of  technical  documentation  available  for 
evaluation;  (4)  are  familiar  to  operational  users  on  both  coasts;  and  (5)  have 
been  accepted  or  implemented  to  some  degree  on  both  coasts. ^  These  two  PMSs 
will  be  briefly  described  below. 

The  Readiness  Estimation  System  (RES)  is  an  automated,  off-line  and 
time-based  measurement  system  under  development  that  continuously  calculates 
the  relative  positional  advantage/disadvantage  of  the  fighter  aircraft  at  a 
specific  point  in  the  engagement.  It  provides  useful  data  for  envelope 
recognition  plus  tactics  and  maneuvers.  Plotted  data  provide  information 
related  to  both  time  history,  and  time  in  envelope  which  is  not  available  on 


^The  Fleet  Fighter  ACM  Readiness  Program  (FFARP)  generates  the  "Blue 
Baron"  reports  that  summarize  individual  squadron  performance  data  taken 
directly  from  TACTS  output.  The  FFARP  is  a  relatively  structured  and 
concentrated  three-week  syllabus  of  ACM  sorties  on  the  TACTS.  It  is  conducted 
annually  by  an  adversary  squadron.  FFARP  exercises,  until  recently,  were 
limited  to  east  coast  TACTS.  The  FFARP  analyses  are  regarded  herein  as  an 
incomplete  system  of  PM,  primarily  because  they  contain  most  of  the 
limitations  of  the  original  on-line  TACTS  ACMPM.  They  do  provide  valuable 
hardcopy  record  of  data  manually  retrieved  from  the  TACTS.  However,  squadron 
feedback  from  VFARP  exercises  is  currently  taking  in  excess  of  two  weeks  and 
the  performance  results  are  costly  to  generate. 
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Che  original  TACTS.  The  RES  was  developed  by  Simpson  (1976)  and  Simpson  and 
Oberle  (19 77). 10 

The  Performance  Assessment  and  Appraisal  System  (PAAS)  is  an  automated 
off-line  system  under  development  to  nrovide  a  systematic.  ACM  performance 
debrief,  including  computer  graphic  formats,  multiple-ref erenced  standards  of 
performance,  diagnostic  analyses  of  ACM  subtask,  and  squadron  performance 
summaries.  It  can  be  described  as  an  informaticn  management  system.  PAAS  Jo 
presented  in  more  detail  by  its  developers  in  a  separate  paper  in  these 
proceedings  (Ciavarelli,  Williams,  &  Brictson,  1981). 

PSYCHOMETRIC  REQUIREMENTS 

Psychometric  requirements  are  the  traditional  and  necessary  ones  that 
establish  the  statistical  soundness  of  a  measurement  system.  They  represent 
an  application  of  the  more  general  requirements  for  any  measurement  system  to 
human  performance  measurement.  Several  specific  requirements  are  identified 
below. 

OBJECTIVITY.  Objective  raw  data  indicators  of  perfor  ■.  -nee,  routinely 
available  from  TACTS,  are  the  foundation  for  the  two  ei  rging  TACTS  PMSs . 
Without  objectivity,  there  is  no  system  of  measurement  because  everyone  is 
free  to  construct  his  own  subjective  "system".  Since  the  TACTS  can  provide 
objective  criteria,  long  .lacking  in  both  selection  and  training  of  military 
aviators,  North  and  Griffin  (1977)  refer  to  the  possibility  of  TACTS  as  an 
"ultimate"  criterion  for  evaluation  of  naval  aviator  selection  variables  in  a 
highly  realistic,  advanced  training  environment.  Similarly,  since  scientists 
are  no  longer  limited  to  instructor  or  peer  ratings  to  assess  training,  some 
researchers  (Ciavarelli  A  Brictson,  1978)  have  referred  to  the  TACTS  PM 
capabilities  as  the  dawning  of  a  "golden  age"  in  operational  training  PM. 

RELIABILITY.  Objective  measures  must  be  reliable.  Reliable  performance 
measures  provide  the  consistency  and  permanence  necessary  for  system 
stability.  Although  there  are  several  methods  for  measuring  reliability,  one 
most  appropriate  for  ACM  training  is  temporal  reliability.  If,  e.g.,  enemy 
aircraft  visual  identification  (VID)  accuracy  is  a  reliable  measure,  a 
particular  pilot  should  be  equally  proficient  (or  non-proficient)  at  the  VID 
task  across  several  recordings  of  VID  performance.  Thus,  reliability  is  the 
basis  for  the  diagnosis  of  consistent  pilot  trends  in  performance.  Diagnosis 
of  individual  pilot  performance  allows  corrective  action  of  substandard 
performance  through  individualized  training.  Unfortunately,  neither  the  FES 
nor  the  PASS  have  documented  the  reliability  of  the  measures  in  their  PMS. 

The  major  limitation  to  reliability  determinations  has  been  the  typically 


l^A  major  modification  of  the  RES,  called  the  Readiness  Index  Factor 
vRIF)  has  been  made  by  operational  aircrew  members  at  the  Oceana  TACTS.  The 
RIF  is  currently  under  evaluation  by  the  NAVTRAEQUIPCEN  as  a  performance 
measurement  tool  for  use  in  a  study  examining  the  transfer  of  training  from  an 
ACM  simulator  (2E6)  to  the  TACTS. 
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small  number  of  repeated  measures  of  performance  available  for  specific 
aircrews  using  the  TACTS. 

VALIDITY.  System  validity  exists  only  if  the  measurement  variables  are 
meaningful.  VID  performance!,  e.g.,  gains  its  meaning  only  through  its 
correlation  with  other  variables,  regardless  of  how  consistent  or  reliable  VID 
performance  might  be.  VID  correlates  with  air  combat  engagement  success,  both 
according  to  tactical  doctrine  (NEWS)  and  empirical  research  tests  of  actual 
performance  on  the  TACTS  (Brictson  et  al.,  1977;  Ciavarelli,  Brictson,  & 

Young,  1979).  Thus,  the  VID  subtask  measure  has  predictive  validity  as  it  can 
predict  ACM  success. 

Concurrent  validity  is  present  when  one  performance  measure  correlates 
with  other  performance  measured  concurrently  (or  nearly  concurrently)  in 
time.  For  example,  a  fighter  pilot  is  twice  as  likely  to  get  a  VID  on  the 
TACTS  if  he  has  gotten  radar  contact  than  if  he  has  not  (Ciavarelli  et  al. , 
1980).  Confirmation  of  such  sequential  probabilities  between  ACM  subtaaks  is 
also  in  agreement  with  tactical  doctrine. 

An  ACMPMS  is  not  completely  valid  without  content  validity,  which  exists 
when  the  performance  measures  are  inclusive  or  at  least  representative  of  the 
ACM  task.  Thus,  a  thorough  job  analysis  of  ACM  is  required  to  ensure  that  the 
performance  measures  selected  represent  the  kinds  of  subtask  content  required 
for  ACM  engagements.  Although  the  RES  has  very  little  predictive  or 
concurrent  validity  documentation  available,  it  is  strong  in  content  validity 
because  it  focuses  its  measurement  system  on  the  maneuvering  task  (ACM)  which 
is  uniformly  regarded, as  the  most  visible  and  dramatic  content  of  air-to-air 
combat . 

SENSITIVITY.  A  measurement  subsystem  is  sensitive  to  other  subsystems  of  the 
overall  training  system.  If  there  is  a  PMS,  it  is,  by  definition  of  a  system, 
interrelated  with  other  subsystems.  A  system  of  ACM  performance  measures, 
therefore,  will  be  sensitive  to  changes  in  other  TACTS  subsystems  such  as 
aircrew,  instructors,  weapons,  etc.  To  illustrate,  the  PAAS  measures  are 
highly  sensitive  to  differences  in  commonly  acknowledged  aircrew  proficiency 
levels  between  fleet  operational  squadrons  and  the  typically  more  experienced 
fleet  reserve  fighter  pilot  squadrons  (e.g.,  Ciavarelli  et  al.,  198C). 
Similarly,  differences  in  thrust  to  weight  ratios  and  wing  loadings  of 
different  aircraft  are  routinely  reflected  in  tee  RES  metric. 

QUALITY  CONTROL.  A  system  of  measurement  does  no:  admit  as  input  data  that 
does  not  meet  certain  quality  of  raw  data  require.,  •  nts .  Only  through 
systematic  filtering  of  input  does  a  system  maintain  its  characteristic  and 
recognizable  features.  Moreover,  such  quality  control  is  necessary  to  achieve 
reliability,  validity,  and  sensitivity  of  the  PMS.  Both  the  PAAS  and  the  RES 
have  quality  control  procedures  for  data  entry  into  the  PMS. 

TRAINING  REQUIREMENTS 

Training  requirements  for  a  PMS  are  those  specifically  appropriate  for  use 
in  training  applications.  They  are  met  most  completely  when  the  above 
psychometric  requirements  have  been  met. 
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DESCRIPTIVE.  A  system  of  measurement  should,  at  minimum,  provide  a  complete 
description  of  relevant  performance.  The  RES,  e.g.,  traces  continuously  the 
maneuvering  skills  of  the  pilot  over  the  course  of  an  engagement. 

EVALUATIVE.  In  order  to  standardize  training,  it  is  necessary  to  evaluate 
performance  in  terms  of  established  standards  of  performance.  Common 
standards  against  which  individual  performance  may  be  gauged  are  a  necessary 
part  of  a  PMS.  At  .least  two  kinds  of  standards  are  required  to  fully  evaluate 
ACM  perlormance.  Criterion-referenced  standards  are  based  on  an  absolute 
reference  point  that  represents  the  criterion  performance  to  be  reached  by 
aircrcv  members.  For  example,  tactical  doctrine  prescribes  missile  envelope 
boundai  Les  outside  of  which  missile  launches  would  be  classified  as  errors. 

It  is  possible  to  compare  directly  tactical  envelopes  (the  criterion)  with 
actual  shots  (placements)  taken  in  TACTS  to  verify  that  3hots  were  within  the 
prescribed  boundaries. 

Since  all  pilots  do  not  always  meet  criterion  on  all  ACM  subtasks,  it 
becomes  necessary  to  describe  the  variability  of  performance,  on  different 
subtasks.  Norm-referenced  standards  describe  the  average  or  normative 
performance,  regardless  of  preestablished  criterion  levels.  Normative 
performance  is  described  essentially  by  using  group  averages  with  a  second 
statistic  representing  variability  of  individual  performance  around  that 
average.  For  example,  tactical  missile  envelope  boundaries  can  be  compared  to 
normative  envelopes  based  on  empirical  data  taken  from  the  TACTS.  One 
practical  use  of  normative  standards  is  to  develop  criterion  standards  from 
them.  DeLeon  (1977),  e.g.,  suggests  that  training  to  gain  incremental 
incre  is.cs  in  air-to-air  combat  skills  above  the  average  skill  levels  is  more 
reasonable  than  trying  to  train  everyone  to  be  an  ace.  Both  kinds  of 
standards  have  been  developed  for  some  subtasks  fcf.,  Ciavarelli  et  al., 

1980).  Ideally,  users  will  soon  be  able,  via  automated  PMS  features,  to 
select  and  create  their  own  performance  norms  for  various  TACTS  operating 
cond  Ltions . 

DIAGNOSTIC.  A  PMS  provides  the  opportunity  for  self-correction  on  the  basis 
of  performance  feedback.  The  system  provides  enough  detail  in  its  diagnosis 
of  tasks  or  subtasks  that  the  individual  can  recognize  those  specific 
behaviors  that  are  not  currently  meeting  established  standards.  The  RES  was 
not  constructed  to  be  diagnostic  of  individual  aircrew  tasks  but,  to  represent 
the  overall  level  of  aircrew  readiness.  The  diagnostic  capability  for  PAAS  is 
juft  now  beginning  to  emerge. 

REMEDIAL.  A  PMS  not  only  informs  an  individual  of  his  skills  (i.e., 
diagnosis),  and  the  level  of  skill  achievement  (i.e.,  evaluation),  it  also 
identifies  specific  corrective  actions  in  the  event  that  remedial  action  is 
necessary.  Neither  PASS  nor  RES  provides  remedial  training  currently. 

TIMELY.  To  maximize  the  self-corrective  feature  of  a  PMS,  feedback  should  be 
us  rapid  and  as  continuous  as  possible.  Because  the  PASS  and  RES  are  off-line 
systems  that  cannot  operate  while  TACTS  training  is  in  progress,  neither  PMS 
provides  immediate  ACM  feedback  after  an  ACM  engagement.  However,  both 
systems  can  provide  extensive  feedback  within  24-48  hours. 
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USER  FEASIBILITY  REQUIREMENTS 

User  feasibility  requirements  are  essential  to  a  PMS  because  if  they  are 
not  met  the  "system"  will  not  be  used.  These  requirements  basically  deal  with 
the  operational  user's  cost,  ease,  and  overall  acceptability  of  operating  the 
PMS.  They  are  even  more  specific  than  the  training  requirements  illustrated 
above  because  they  apply  to  ACM  training  in  particular  and  the  feasibility 
issues  associated  with  that  specific  application. 

AFFORDABLE.  Ac  this  point,  both  the  PAAS ,  and  particularly  the.  RES,  require 
an  analyst  to  process  and  interpret  the  performance  feedback  from  the  PMS 
Before  either  PMS  gains  long-lasting  use,  it  will  be  necessary  to  minimize 
those  costs  which  support  the  system. 

ADAPTABLE.  Changes  induced  by  system  growth,  and  changes  in  user  needs, 
including  changes  in  training,  must  not  disrupt  the  basic  operation  of  the 
PMS.  Design  modularity  is  essential.  Thus,  it  should  not  be  essential  to 
continue  measuring  out-of-date  variables  in  order  to  add  a  new  and  necessary 
measurement  variable.  Likewise,  it  should  not  be  essential  to  eliminate  still 
useful  variables  in  order  to  drop  an  old  and  now  unnecessary  variable.  The 
PASS,  and  to  a  lesser  extent,  the  RES,  are  modularized  so  that  they  will  nor 
lose  their  system  status  due  to  system  growth. 

AUTOMATED.  Much  of  the  data  currently  available  to  users  of  the  TACTS  are 
only  available  with  a  great  deal  of  manual  extraction  efforts.  Since 
instructor  and  student  workload  can  prevent  use  of  otherwise  valuable 
performance  feedback,  both  the  RES  and  PASS  are  automated. 

MANAGEABLE.  Designers  of  the  PMS  should  reduce  the  number  of  feedback 
variables  to  those  required  for  meeting  behavioral  training  objec-ives.  The 
data  flooding  referred  to  earlier  is  evidence  of  the  confusion  that  a  PM. 
system  serves  to  reduce.  The  PASS  and  RES  share  a  significant  advantage  in. 
this  respect. 

ACCESSIBLE.  For  performance  data  to  be  used,  they  must  be  easily  accessible 
by  the  user  subsystem  that  interacts  with  the  PMS  subsystem.  Ae  ideal  PMS 
makes  it  rather  simple  for  users  to  both  enter  and  retrieve  performance  data. 
The  PAAS,  in  contrast  to  the  RES,  was  designed  especially  with  accessibility 
in  mind,  by  incorporating  a  r '•.ruber  of  "user  friendly"  computer  features. 

NON-INTERFERENCE  WITH  TRAINING.  3oth  PM.Ss  are  currently  off-line  because 
their  or  line  operation  would  require  dedicated  TACTS  computer  time  which 
would  c  .crease  the  amount  of  already  scarce  ACM  training  time.  Although  the 
ultimate  goal  is  to  gain  on-line  PMS  capabilities,  off-line  processing  js  the 
only  currently  feasible  alternative. 

ACCEPTABLE .  Acceptability  is  an  umbrella  term  for  all  user  requirements  that 
determine  whether  TACTS  users  will,  in  fact,  be  motj  'ated  to  use  the  PMS,  For 
example,  the  RES,  a  PMS  with  a  great  deal  of  techni  merit,  has  undergone 
major  modifications  at  Oceana,  partly  because  it  was  not  easily  understood  or 
used.  Without  user  acceptability,  even  a  well-validated  and  low  cost  PMS  has 
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no  value.  Thus,  certain  ACMPMS  features  that  have  more  to  do  with 
motivational  than  instructional  capabilities  per  se  are  required  for 
acceptability.  Motivational  factors  that  impacc  system  utilization  include 
user  choice  of  alternative  graphic  display  formatting  of  performance  feedback, 
possibility  for  instructor  override  and  flexible  use  of  automated  PMS, 
administrative  record  keeping  assistance  to  the  instructor  by  the  PMS,  privacy 
coding  of  performance  results,  etc. 
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SECTION  IX 

CONCLUSIONS:  TACTS  TECHNOLOGY  AND  HUM4N  PERFORMANCE  MEASUREMENT 


Without  question,  the  TACTS  is  an  impressive  application  of 
state-of-the-art  technology  in  airborne  instrumentation  and  tracking, 
computer-generated  simulation  of  weapon  launch  outcomes,  video  tape  playback 
of  voice  transmissions  and  graphic  portrayal  of  flight  history  data.  Equally 
impressive,  however,  is  the  conclusion  that  the  TACTS  was  developed  and  is 
currently  operated  without  the  benefit  of  a  system  of  performance 
measurement.  This  conclusion  is  based  on  several  observations:  (1) 
identification  of  several  limitations  in  the  TACTS  capabilities  for  training; 
(2)  a  recent  (March  1981)  naval  message  from  the  FPT  for  air  combat  training 
that  an  ACMPMS  was  needed;  (3)  recent  attempts  by  operational  users  of  the 
east  coast  TACTS  facility  to  develop  their  own  PMS;  and  (4)  a  mismatch  between 
the  original  TACTS  PM  scheme  and  a  set  of  psychometric,  training,  and  user 
feasibility  requirements  for  a  PM  system  (PMS). 

In  terms  of  the  three  original  requirements  leading  to  development  of 
TACTS,  the  third  requirement,  for  safety  and  economy  of  training,  has  been 
achieved.  Fulfillment  of  the  second  requirement,  for  a  performance  debrief 
capability,  depended  in  large  part  on  the  development  of  a  PMS  and,  therefore, 
has  been  only  partially  achieved.  The  first  requirement,  for  missile  envelope 
recognition  training,  is  difficult  to  evaluate  because  there  is  not  one 
envelope,  but  multiple  envelopes  to  be  learned  for  different  missiles  and 
aircraft  capabilities.  In  addition,  there  is  no  completely  uniform  agreement 
as  to  the  "proper"  firing  envelope  for  a  specific  missile.  Thus,  it  is  not 
nurp-’ising  that  several  systematic  samples  of  launch  data  obtained  on  the 
t ACTE  indicated  that  even  experienced  pilots  often  fire  outside  of  prescribed 
doctrinal  missile  envelope  limits  (Ciavarelli,  Narsete  4  Brictson,  1981). H- 
Such  'variability  in  performance  is  in  part  due  to  the  lack  of  a  PMS  for 
standard!:  'ion  of  envelope  recognition  training . 42 


•'■■'■A  major  advantage  of  TACTS  is  that  it  allows  such  systematic 
evaluations  of  envelope  recognition  skills. 

^A  second  major  reason  is  that  TACTS  users  do  not  generally  use  the 
five  available  envelope  recognition  training  modes  which  vary  progressively  in 
level  of  difficulty  and  amount  of  feedback  provided.  The  strong  preference  is 
to  use  mode  five  which  is  the  most  realistic  and  challenging  one.  A  third 
significant  reason  is  that  the  TACTS  provides  very  little  envelope  recognition 
practice  for  any  particular  aircrew.  Access  to  the  range  is  highly 
competitive.  During  TACTS,  the  engagements  are  usually  no  more  than  several 
for  each  aircrew  and  engagements  typically  last  less  than  a  few  minutes. 

Users  (e.g.,  Carter,  1979)  point  out  that  the  recognition  skili  is  "achieved 
only  through  repetitive  exposure  to  possible  firing  situations  (p.  68)".  An 
analytical  effort  prior  to  TACTS  development  to  identify  recognition  training 
capabilities  would  almost  certainly  have  concluded  that  simulator  training 
would  be  needed  to  supplement  practice  on  the  TACTS. 
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The  theme  for  the  1981  Symposium  on  Aviation  Psychology  asked,  "Is 
advancing  technology  ignoring  human  performance  in  aviation  systems?"  It 
would  appear  that  with  the  development  of  the  TACTS  instrumentation 
technology,  there  was  no  parallel  development  to  provide  a  PMS  to  support,  that 
capability.  This  apparent  "oversight"  is  a  little  easier  to  understand  when 
one  considers  the  almost  mind-boggling  complexity  of  the  ACM  task  and  the 
fighter  mystique  which  is  the  cultural  heritage  associated  with  ACM.  In 
addition,  during  production  of  the  original  TACTS,  there  was  not  available  a 
well-developed  PMS  that  could  have  been  further  developed. 

It  was  suggested  that  there  are  several  viable  approaches  to  the 
measurement  of  TACTS  performance.  Two  promising  and  emerging  systems  of  ACMPM 
were  described  and  used  to  illustrate  a  set  of  functional  requirements  for  a 
PMS.  Fulfillment  of  these  requirements  will  provide  valuable  performance 
criteria  for  a  number  of  different  purposes,  including  maximization  of  the 
training  capabilities  of  the  TACTS,  and  will  allow  the  "golden  age"  of 
operational  performance  measurement  to  mature. 
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